The purpose of an artificial neural network is to imitate the functioning of a biological nerve system. Research in neural networks is focused on developing a system with a high degree of parallel processing capability to more efficiently solve complex problems as compared to normal computer systems. The speed and robustness of neural networks makes them attractive for applications such as pattern and speech recognition.
A network is usually characterized by simple functional units, each consisting of a neuron and a synapse or synaptic cell. Each synaptic cell provides storage of either a fixed or adjustable synaptic weight, and has an output which is a function of the input multiplied by that weight. The ability to change this weight or strength of the synapse enables the network to handle incomplete or noisy date and adapt itself to the near-correct solutions.
A biological neuron is typically connected to several thousands of other neurons through the synapse cell which result in high degree of parallelism. Due to the massive connectivity, a large number of synapses and neurons are required in hardware implementation and in turn, area limitation becomes the most crucial factor in its implementation.
It is a feature of this invention to provide a synapse for neural networks which enables the number of required external input of output stages, and the number of interconnections within the neural network, to be minimized, thereby creating a dense neural network with optimum utilization of chip area. The invention therefore provides a synapse having a small architecture requiring internal interconnectivity optimal for artificial neural networks.
Often neural network applications require several bits of resolution in synaptic weight which needs excessive chip area in digital implementation. Analog implementation on the other hand requires much smaller synaptic cells for the same resolution as that of digital with less supporting circuitry and interconnections, and are thus more attractive by requiring less IC area than digital implementations.
Several analog implementations of a neural network synaptic cell have been developed. One approach is using fixed resistors to implement a given synaptic weight cell; however, a network of such cells is static, since it lacks the learning capacity provided by adjustable weight levels. Also standard manufacturing technology makes it difficult to achieve precise value of resistance for weights. A second approach is the use of conventional multiplication (multiplier) circuits to create a synapse. Such multiplier circuits require excessive chip area due to the large number of transistors. A third approach is the storage of the synaptic weight as a charge in one or two capacitors. However, capacitors require a relative large area to implement on an IC, and is volatile due to current leakages; thus, capacitors require periodic refreshing to maintain precise weight storage. A fourth approach uses both digital and analog hardware, by having a RAM cell for weight storage and an analog circuit for converting the weight into a current. This synaptic cell approach also requires extensive area usage due to the RAM cell, and additional circuitry for digital to analog weight conversion.
Other types of analog synapses have used floating gate MOSFETs which have the capability of non-volatile weight storage by storing a charge in the floating gate by tunneling injection, which subsequently shifts the threshold voltage level of the device. The floating gate technology is based on tunneling effect described in an article entitled Fowler-Nordheim Tunneling into Thermally Grown SiO.sub.2, by M. Lenzinger and E. H. Snow, Journal of Applied Physics, Vol. 40 No. 1, January 1992. However synapses incorporating these devices each requires additional area for both circuity and interconnections which the present invention avoids.
An analog synapse with such MOSFETs uses one floating gate MOSFET with four other FETs in a transconductance amplifier configuration (See the article entitled Analog Floating-Gate Synapses for General-Purpose VLSI Neural Computation, by Bang W. Lee, et al., IEEE Transactions on Circuits and Systems, Vol. 38 No. 6, June 1991). This synapse however requires that the output of a neuron be converted from a voltage to a current prior to being inputted into a synaptic cell, which is provided by the addition of a two transistor circuit. A programmable synapse based on two floating gate MOSFETs in a common-input differential-output configuration is described in an article by A. Krammer, et. al., entitled Compact EEPROM-based Weight Functions, 1990 Neural Information Processing System Conference Proceedings, Nov. 7, 1990. This differential output approach requires that the two output currents be summed by a current comparator external to the synaptic element to provide a single output current. A programmable synapse similar to that described by Krammer, but requiring an additional transistor to function is described in an article by Borgstrom, entitled Programmable Current-mode Neural Network for Implementation in Analogue MOS VLSI, IEE Proceedings, Vol. 137, Pt. G., No. 2, April 1990. Such circuitry which requires either an amplifier or differential output circuitry structure, mitigates against the optimal utilization of IC semiconductor area.
Holler et. al. discusses the use of this MOSFET technology in U.S. Pat. No. 4,956,564 issued Sept. 11, 1990 and entitled "Adaptive Synapse Cell Providing Both Excitatory and Inhibitory Connections in an Associative Network". The patent shows a two quadrant multiplying synapse which has two floating gate MOSFETs in a single input differential output configuration, and a four quadrant multiplying synapse which has four floating gate MOSFETs in an array in a differential output configuration with dual complementary input lines. Each synapse circuit requires two capacitors, one on each output line, that discharge when the MOSFET conducts a current. However, due to both the differential output circuit structure, and their dependency on capacitance to generate current on the output lines, neither synapse is optimal in interconnectivity and semiconductor area utilization.
U.S. Pat. No. 4,999,525 issued Mar. 12, 1991 to Park et. al. and entitled "Exclusive-Or Cell For Pattern Matching Employing Floating Gate Devices" shows four floating gate MOSFETs in an array configured along two voltage input lines, where one is the complement of the other, and two differential output current lines. Such a synapse cell does not have optimal architecture for IC implementation, specifically in the required number of interconnections due to both its transistor array and differential output structure.
Further both Holler and Park patents show a synapse with a differential output circuit requiring a current comparator for summing the two output currents, which is an additional circuit which is not required by the present invention. Thus, both patents show a four quadrant multiplier synapse using four floating gate MOSFETs which requires less than optimal IC configuration. It is a feature of the present invention to optimize interconnectivity and minimize the cell area, while providing four quadrant operation.
Therefore, the analog synaptic cells using floating-gate MOSFETs described in the above discussed literature and patents have several drawbacks including the extent of the semiconductor area required for an analog synapse in VLSI technology, in that the differential output requires two output lines and a separate current comparator to sum that differential output, or an amplifier structure using five or more transistors and an input voltage to current stage. The need for a current comparator results in the excessive area usage since there are two summing output leads for each synapse which need to be routed to a comparator circuit to generate a single output. This comparator circuit must be either a part of or prior to each neuron, and as stated in the Borgstrom article such a current comparator would require the addition of an eight transistor circuit stage. Even though the amplifier configuration for a synapse avoids a differential output and current comparator circuit, it achieves this at the expense of additional transistors, internal cell circuity, and an input current to voltage converter stage at the output of each neuron, and as such, does not minimize the use of semiconductor area in a VLSI chip. It is a feature of the present invention to provide a synapse which has a single output line and does not need a current comparator circuit or additional transistors.
Synapses, in accordance with the Holler patent, require a capacitor on each output line. VLSI technology requires a relative large area to construct such capacitors, and are costly to create within precise capacitance tolerances. Therefore it is a feature of this invention to provide a synapse which does not need a capacitor on the output line, and still has a minimal number of required interconnections, while maintaining all the functions needed in a synapse.
A further drawback of most of the analog synapses, in the above discussed documents, is that they are limited to two quadrant multiplication. In other words, the output current of the synapse versus the input voltage is limited to the first and third quadrants. To most accurately represent a biological neural system, full four quadrant multiplication is needed to allow for all levels of inputs and outputs. Two quadrant multiplication alone limits the ability of the network to model a natural neural network that a four quadrant multiplying synapse would afford. Although Holler and Park patents show a four quadrant multiplier, their synapses require area for circuitry and interconnections that the present invention avoids. This is due to their four floating gate MOSFETs, array circuit structure, dual input and output lines, and the requirement of a current comparator to sum the differential output. Further, the patents fail to disclose possible implementation of four quadrant multiplication for an analog synapse. In addition, the need for a circuit comparator at the output requires that the output signals pass through additional active components which negatively affects circuit linearity. It is a feature of the present invention to provide a synapse which requires less area to implement than Holler or Park, while providing improved linear response over four quadrants.